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00:00 Introduction 


Evaluation is a critical part of developing high quality public engagement and in this short 
webinar we'll be looking at the key ways that you can develop a really effective evaluation 
approach for your work. 


My name is Sophie Duncan and | work at the National Co-ordinating Centre for Public 
Engagement, and | am really passionate about the role of evaluation in informing what we 
do and in evidencing if and how it has value. 


To start with | would like to say that evaluation is a friend of anyone who wants to do high 
quality public engagement, but it is a critical friend, and you will sometimes find out things 
through evaluation that may disappoint you or may make you have to think really really 
hard about how your activity is fit for purpose. However, evaluation helps you work out how 
to make your project fit for the purpose and fit for the participants. 


Evaluation is part of planning a project and isn't something that you should be leaving till 
the end or to the point of intervention, so often at the point of intervention people think, 
‘oh, | haven't thought about evaluation’ and they immediately want to evaluate the 
participants. It's much better to think about evaluation strategically. Think about where you 
use your resource what techniques you use to gather your data and really think hard about 
what you're trying to find out through your evaluation and what you're going to do with that 
evidence. And finally, it really helps you work through that iterative cycle that leads to 
quality engagement. 


01:42 Four pillar approach 


In order to think about the fundamentals for designing a really high-quality engagement 
plan, I'm going to draw on the NPC's four pillar approach, and we have a link to this resource 
on our website. The four-pillar approach starts with a foundation of leadership and 
commitment for evaluating well. It then has four pillars on which to build an effective 
evaluation approach, and I'm going to take each in turn. 


The first pillar is to map your theory of change. And a theory of change is simply how you 
think change is going to happen: how the activities you're planning in your project are going 
to lead to the change that you're seeking. You can do a theory of change in lots of different 
ways, but today I'm just going to focus on one way of presenting a theory of change, a logic 
model. 


02:26 Theory of change and logic models 


At a very basic level a logic model just helps you spell out the different elements of your 
project, from the inputs — what you're putting in. So, that might be your time, it might be 
your partner's time, it might be resources, it might be funding. It also then gives you a 
chance to describe well what are you going to do: what are the activities that you're going 
to do as part of your project. And finally, what are you going to achieve through those 
activities: what kinds of outcomes will there be. 


This next diagram is a slightly more complicated version or more detailed version of the 
basic logic model | showed in the previous slide. And there’s a copy of this on our website. 


Here you'll see that there's some additional categories. So first and foremost, what's the 
purpose of your activity? What's its main aim and who are the participants? And that might 
be the publics who are involved, the community organizations are involved, the partners 
that you are working with. 


You will see that again we have inputs, what you're going to put in and the activities what 
you're going to do. But now we've created three columns — outputs, outcomes, and impacts. 
It's fair to say that the outputs are basically just that: what's come out of the project. You 
might have created a website; you might have created a resource; 50 people might have 
attended an event. On the whole you can quantify outputs relatively easily and also you can 
assess that data and we often call that monitoring data — data that you can just find out by 
looking at your project and what you've done. Outcomes and impacts are what's resulted as 
a result of your project, and we've chosen to split them into two just to help us think about 
the immediate outcomes or the immediate impacts of a project. So, what's been achieved at 
the point of intervention and then those longer-term impacts that perhaps happen over a 
longer period of time, and as a consequence may take a different approach to thinking 
about how you would measure. 


You'll see also that we've got a box on assumptions, and these are the assumptions that 
you're making in developing your theory of change. And we make lots of assumptions about 
how the activities that we are designing are going to actually lead to the impacts that we 
hope for. And then there's factors and risks, things outside of your control that actually 
could impact your program. And finally, there's evaluation, but I'll be talking about that 
shortly. 


Creating a model of your theory of change is a really good collaborative activity to do if 
you're working with a team or you're working in partnership. It helps you think collectively 


about the activities you choose to prioritize in order to lead to the kinds of outcomes and 
impacts that you care about, and it also helps you to check in with one another about the 
assumptions you have on the program. 


05:29 The second pillar 


The second pillar is to prioritize what you measure, and this is all about the purpose of the 
evaluation and why you're evaluating and who you're evaluating for. Prioritizing what you 
measure is a critical part of deciding your plan and sometimes because we leave it to too 
late in the process, we end up directing our evaluation effort at the wrong point in the 
program. 


It's worth remembering that evaluation can be used at the beginning, throughout, and at 
the end of a program to address the evaluation questions that you have. Front-end 
evaluation helps you to gather insight and learning that can actually help you develop your 
project. Formative evaluation helps you as you go through the process to reflect on what's 
working well, what isn't working so well, and if and how the interventions you're designing 
or piloting are actually working. And finally summative evaluation helps you to look back on 
the program to see what worked, what learning you have, and perhaps some of the things 
that you might think about for future programs. 


When thinking about what you want to know you may find this guide helpful. This guide is 
from Better Evaluation and they're looking at the kinds of areas that evaluation questions 
could explore. The first is descriptive questions. So, they are questions that help you think 
about what happened. And there are causal questions that look at what's contributed to the 
changes that have been observed and the role of your intervention or your program within 
that. There are process questions about whether the process could be considered a success 
and who was it a success for. And action questions that look at what we're learning from the 
program and what should be done to respond to the evaluation findings. 


This slide is available on our website for you to find out more. 


It's really important to consider those questions that you have for the evaluation, and those 
larger kind of evaluative questions. It's really helpful to do that with your collaborators and 
your partners to consider what's important to you to find out from your evaluation. And as a 
rule of thumb, around six main overarching kind of questions to explore is probably enough. 
And once you've got those questions you should map them onto the logic model because it 
will depend on where you do the evaluation to source the evidence required to answer 
those overarching questions. So, it might mean that you need to do a baseline before you 
start your intervention, or it might mean that you need to do some formative evaluation 
throughout your program. And when you know what you're trying to find out it gives you a 
really good idea then about where and how you're going to get that data. 


08:15 The third pillar 


The next pillar is all about choosing your level of evidence. And this is really about what's 
the purpose of your evaluation and that will determine the level of evidence that you might 
want to seek. Three important things to think about as shown on this slide. 


The first is about proportionality and the extent to which the energy and effort and resource 
you're putting into your evaluation is proportionate to the energy, resource, and effort 
you're putting into your project. It's fair to say that there isn't a strong rule of thumb here 
because it really depends on the purpose of your evaluation. And you can imagine that if 
you're doing a pilot program you might want to put more of a percentage of your resource 
into that evaluation because it's really important to assess what's working well, what needs 
to change, and how you might then scale that up to a larger program, and so therefore the 
evaluation data is going to be critical in helping you design that larger project. Whereas if 
you're running an event that happens on a monthly basis and has a similar format on that 
monthly basis then you may, after evaluating the first few events, choose only to evaluate a 
sample of the future events because you've already gathered enough knowledge and 
understanding about what works and what doesn't work to inform what you're doing. 


The second thing to think about is the use of the evaluation and who you're evaluating for. 
And if you're evaluating for you and your team you may require a different level of evidence 
than if you're for example evaluating for a Research Excellence framework case study where 
you're trying to evidence the impact of the engagement work that you've done. 


The final thing to think about is the quantitative and qualitative methods and balancing 
those in how you then think about the evidence you present. Quantitative data is really 
good in terms of getting scale. “Yeah 80 of the 100 attendees all learned something new.” 
And qualitative data really helps you understand more about what's going on. So, if 80 had 
learned something new then the qualitative data will give you a sense of what they've 
learned that is new. 


10:16 The fourth pillar 


The fourth pillar is all about selecting your sources and tools and there are lots of tools to 
choose from and each of them have pros and cons. Here are some examples of the kinds of 
tools that you might use, and on our website, we've got more information about some of 
these tools and the pros and cons of their use. 


It's worth saying that it's really important to think about the participants who you're going 
to involve in the evaluation and choosing tools that are appropriate for their needs and 
interests. And where possible it’s a really good idea to try and embed evaluation into the 
intervention rather than have it as an add-on at the end. 


Some of these examples include things like physical ranking scales using people; or 
questionnaires that you might do; there’s personal meaning mapping, where perhaps 


individuals then try and make meaning from what they've engaged with, and that's 
particularly useful if you're working with people over a longer period of time; a graffiti wall’s 
a quick and easy way to capture feedback at an event around a particular question, where 
people can write on a blackboard or stick up a Post-It to address a question that you're 
asking; and there are things like observations, where perhaps you work with an 
independent evaluator who observes what's going on in order to reflect back into the 
evaluation. 


Some of the key questions to ask when you're planning your approach is the suitability of 
the intervention or the technique for the participants in the evaluation. The second thing to 
think about is how you're going to analyse the data. So, the resulting data — do you have the 
skills to analyse it, or do you have the skills across your team to analyse it? And also do you 
have the time? So, we might want to use a video booth to capture people's feedback during 
a live event, but do we have the time to watch those videos to transcribe and analyse the 
responses that were made? 


There are clearly ethical considerations when doing evaluation; and those ethical 
considerations depend on who you're involving in the evaluation, what you're asking, and 
the data you're acquiring from people, and also what you're going to use that data for. You 
might want to consider sampling, so actually do you try and ask everybody at an event to 
respond to your evaluation or you're going to use a sampling strategy. And determining that 
in advance is really important. And finally, you might want to think about coding, so if you're 
gathering qualitative data how might you code that data and what techniques might you 
use to do that? 


And you'll be pleased to know that there are coding frameworks available, and one that's 
particularly relevant to public engagement is the generic learning outcome framework, the 
Glo framework, that was developed by the Museum, Libraries, and Archives Council to 
support museums, libraries, and archives to assess the quality of experience gained by 
visitors to their space. 


And you'll see that they have five categories that include things like knowledge and 
understanding, so have people learned something new. And also enjoyment, surprise, and 
creativity: how have their experience of the event? 


Once you've gone through those four pillars — you've mapped out your theory of change; 
you've prioritized what you're going to measure so you understand the purpose of your 
evaluation and what you're trying to find out; you've chosen your level of evidence; and 
you've selected your sources and tools — you've developed basically a framework for your 
evaluation. And you have a much clearer sense about what you're trying to achieve. 


13.39 Evaluation plans 


Before | sign off, | just want to tell you a little bit more about what might go into an 
evaluation plan, and you'll be pleased to know an evaluation plan is just an extension of the 
four pillar approach. So, you need to include your aims, your objectives, how you're going to 


achieve those aims, your outputs, outcomes, and impacts, and all of those are already 
included in the logic model we showed you earlier. 


You then need to know your overarching evaluation questions and you've figured those out 
when you've been thinking about prioritizing what you're going to measure. 


You need to then think about your methodology, the strategy you're going to use, the data 
collection and the data analysis. Ad that's all about what you've explored in working out 
your level of evidence and the tools and approaches that you're going to use and how 
you're going to analyse that evidence. 


And the final thing you need to think about in your evaluation plan — we haven't really 
touched on in the four pillar approach — but is that reporting. What kinds of outputs are you 
going to produce from your evaluation and for whom? Are you going to have a report that 
goes to your funders? In which case really helps to understand what they'll be looking for 
from that output. You might want to consider if you want to develop an output for 
participants in the program that might capture some of the learning that they've provided 
for you about what worked well and what didn't. 


| really hope that this webinar has given you a taste of how to develop your own evaluation 
approach. And there are lots of resources available to help take you through the 
practicalities of doing that, many of which we've pulled together on our website. | really 
look forward to seeing how your evaluations develop and to hearing about the learning that 
you've had through evaluating your work well! 


